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ABSTRACT 

Optimal  control  of  systems  governed  by  delayed-dif ferential 
equations  is  explored  by  using  the  control  theorv  developed  for 
systems  governed  by  ordinary  differential  equations.   A  simple 
algorithm  for  producing  a  suboptimal  control  law  with  restricted 
feedback  is  presented.   Two  examples  illustrate  the  computational 
method . 


FOREWORD 

This  report  summarizes  a  research  effort  which  has  been 
funded  by  the  Foundation  Research  Program  of  the  Naval  Postgraduate 
School,  Monterey,  California. 
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I.   INTRODUCTION 

A.  DELAYED-DIFFERENTIAL  EQUATIONS 

In  many  physical  systems,  such  as  rocket  engines  [1,2], 
antirolling  stabilization  systems  for  destroyers  [3]  and 
pipeline  recycling  systems  for  chemical  reactors  [*4],  the 
most  accurate  mathematical  representation  of  system 
dynamics  is  found  to  be  a  matrix  system  of  differential 
equations  with  finite  time  delay  in  the  arguments  of  some  of 
the  state  variables. 

x(t)  =  Ax(t)  +  Bx(t-x)  +  Cu(t)  (1) 

t  >  0    ,    T  >_  0 

x(t)  =  4>(t)  ;   -t  <  t  <_  0  (2) 

Equations  of  this  form  have  been  known  as  differential- 
difference,  hysterodif ferential,  functional  differential, 
differential  with  transport  lag,  differential  with  deviating 
arguments,  and  delayed-dif ferential  equations,  to  name  just 
a  few.   For  the  purposes  of  this  research,  equations  of  the 
form  of  (1)  are  referred  to  as  delayed-dif ferential 
equations . 

B.  THE  OPTIMAL  REGULATOR  PROBLEM 

The  research  to  be  described  is  concerned  with  a  solution 
of  the  optimal  regulator  problem.   Simply  stated,  this 
problem  seeks  the  control  function  u*(t)  (a  member  of  the 
set  of  admissible  controls  u(t))  which  minimizes  a  quadratic 


cost  function  of  the  form 


J(<Kt),x(t),u(t))  =  /   [x'(t)Qx(t)  +  u'(t)Ru(t)]dt  >  0  (3) 

0 


subject  to  the  constraints  of  equations  (1)  and  (2).   Q  is  a 

2  2 

n  positive  semidef inite  matrix;  R  is  a  I     positive  definite 

matrix;  I   <   n  . 


C.  NOTATION 

Notation  follows  that  established  by  Bryson  and  Ho  [5]. 
In  general,  it  is  as  follows: 

1.  Column  vectors  are  denoted  by  lower  case  letters. 
Single  subscripted  lower  case  letters  are  elements  of 
vectors. 

2.  Matrices  are  indicated  by  upper  case  letters.   Doubly 
subscripted  lower  case  letters  represent  elements  of 
a  matrix. 

3.  Matrix  transpose  is  indicated  by  the  prime  symbol(f). 
The  following  are  deviations  from  the  general  rules 

established  above. 

1.  t  represents  the  scalar  independent  variable  time. 

2.  t  denotes  the  scalar  time  delay,  possibly  a  function 
of  time  and  state  variables. 

3.  J  is  a  scalar  function  of  matrices. 

4.  The  lower  case  letters  i,j,£,m,  and  n  are  used  as 
indexing  variables. 

5.  The  total  derivative  is  indicated  by  d. 

6.  A  damped  natural  frequency  is  represented  by  u) . . 

j 

D.  PROBLEM  STATEMENT 
1.   Introduction 

Inclusion  of  the  delay  term  x  in  the  state  variable 
equation  greatly  complicates  solution  of  the  optimal 
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regulator  problem.   The  well  established  theory  of  ordinary 
differential  equation  systems  can  not  be  applied  with 
equation  (1).   In  addition,  although  the  necessary  theory 
and  computational  procedures  have  been  developed  [6]  for 
computation  of  the  optimal  control  law,  implementation  of 
that  law  is  often  found  to  be  either  impractical  or  undesir- 
able.  Implementation  of  the  optimal  control  law  requires 
feedback  of  a  continuum  of  states,  x(c),  t  -  t  <_  c  _<  t  , 
which  are  often  not  readily  available  from  the  system  being 
controlled. 

2 .   Purpose 

The  purpose  of  this  research  is  to  develop  a  method  ■ 
for  approximating  the  optimal  control  law  for  delayed- 
differential  systems  by  utilizing  the  optimal  control 
formulation  for  ordinary  differential  systems.   In  addition, 
a  simple  algorithm  is  offered  for  restricted  feedback  cases. 

E.   BACKGROUND 
1.   General 

Anderson  and  Soudack  [7]  indicate  that  the  study  of 
the  delayed-differential  equation  appears  to  have  started 
with  John  Bernoulli  in  1728,  but  most  presently  available 
information  has  been  published  since  1940.   Bibliographies 
published  by  R.  Weiss  [8]  in  1959  and  Chosky  [91  in  I960 
contain  references  to  more  than  350  separate  papers  dealing 
with  the  delayed-differential  equation  and  associated 
problems.   The  annotated  bibliography  by  R.  Weiss  surveys 
English  language  papers  published  between  1935  and  1958. 


The  work  by  Chosky  is  supplemental  to  the  bibliography  of 
Weiss  and  covers  the  period  of  1916  to  1959,  including 
reference  to  numerous  foreign  language  papers. 

Prior  to  i960,  several  papers  were  published  which 
addressed  the  problem  of  approximating  the  delay  term  so 
that  the  desired  control  law  could  be  obtained. 

In  a  paper  published  in  19^0,  Mason  and  Philbrick  [10] 
devised  a  liquid  level  analog  to  a  thermal  process  with  time 
lag.   This  model  was  used  to  examine  response  to  various 
types  of  control  systems.   A  related  work  by  Ziegler  and 
Nichols  [11]  considered  various  control  settings  for  auto- 
matic control  circuits  with  process  lag.   The  optimal  control 
was  determined  from  the  area  under  a  state  time-history 
curve.   Bretoi  [12]  and  Minorsky  [3]  used  first  order  differ- 
ential equations  to  approximate  control  systems  with  time 
lags.   After  the  control  law  was  determined  from  the 
approximation,  the  actual  system  was  adjusted  to  give  the 
desired  results. 

In  an  attempt  to  explain  why,  under  certain  circum- 
stances, a  pendulum  began  to  spontaneously  oscillate  with  a 
frequency  higher  than  its  own  damped  natural  frequency  or 
that  of  the  externally  applied  forcing  function,  Minorsky  [13] 
expressed  the  system  model  as 

xx(t)  +  a1x1(t)  +  b^Ct-x)  +  c1x1(t)  =  0     (4) 

Considering  the  time  delay  term  x  to  be  small,  x,(t-x)  was 
expanded  in  a  Taylor  series  to  yield 


,n  in  „fn+l) 


(-D1  ^r  *i   '(t)  +  ...  -  xx1(t)  +  X;L(t) 


For  very  large  n  and  small  t  this  expression  converges  to  a 
term  of  the  form  e~"  .   Replacing  the  delayed  term  in  equation 
(4)  by  e~  the  following  system  model  was  obtained: 


x1(t)  =  (ax  -  b1e"a)  x1(t)  +  c^Ct) 


This  equation  was  studied  to  determine  regions  of  stability. 

The  major  work  in  the  theory  of  delayed-differential 
equations  has  been  undertaken  since  I960.   This  increasing 
emphasis  may  in  part  be  attributed  to  the  rising  application 
of  such  equations  to  automatic  control  systems  as  well  as 
the  development  of  high  speed  computers. 
2.   Development  of  Theory 

Bellman  [lH]  and  El'sgol'ts  [15]  discussed  the 
existence  and  uniqueness  of  solutions  to  equations  of  the 
form  of  (1) . 

Krosovskii  [16],  appears  to  have  been  one  of  the 
earliest  developers  of  an  optimal  control  formulation  for 
the  linear  quadratic  problem  with  time  delays  in  the  states. 
Although  he  gave  no  explicit  definitions  for  the  parameters 
required  to  actually  determine  the  optimal  control  law;  the 
feedback  was  in  the  form  of  a  linear  combination  of  the 
states.   Extension  of  Pontryagin's  maximum  principle  to 
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systems  with  multiple  delays  was  accomplished  by  Kharatishvii 
[17,18].   These  works  have  served  as  a  basis  for  numerous 
other  papers  on  differential-difference  equations.   Orguz- 
toreli  [19],  Chyung  and  Lee  [20]  have  also  worked  with  the 
derivation  of  necessary  conditions  for  optimality  in  the 
form  of  the  maximum  principle. 

Ross  and  Fliigge-Lotz  [6]  developed  an  optimal  linear 
control  law,  similar  in  form  to  Krosovskii's,  for  a  system 
in  which  the  time  delay  had  been  normalized  to  one.  (In  a 
later  work,  Ross  [21]  showed  that  a  system  with  any  number 
of  delay  terms  can  be  transformed  into  a  system  with  one 
delay  which  can  be  normalized  to  one.)   This  paper  also 
presented  the  sufficient  conditions  for  the  existence  of 
admissible  control  functions  and  sufficient  conditions  for 
a  linear  control  law  to  be  optimal.   Kushner  and  Barnea  [22] 
showed  that  a  linear  control  law  for  (1)  subject  to  a  qua- 
dratic cost  function  in  the  form  of  (3)  was  optimal  with 
respect  to  the  class  of  square  integrable  control  laws,  if 
the  solution  to  (1)  was  continuous. 

3.   Approximation  of  Delayed-Differential  Equations 

Because  of  the  complexity  of  solving  the  optimal 
regulator  problem  when  the  system  dynamics  involve  delayed- 
differential  equations,  considerable  effort  has  been 
directed  toward  approximating  the  delayed-dif ferential 
equation  by  systems  of  ordinary  differential  equations. 

Repin  [23]  has  shown  that  equation  (1)  can  be 

approximated  to  any  desired  accuracy  by  a  series  of 

differential  equations. 
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For  systems  in  which  the  delay  term  is  small,  Jen- 
Wei  [24]  developed  a  procedure  for  replacing  the  delay  term 
by  a  series  of  piece-wise  linear  functions  which  lead  to 
a  system  of  linear  state  variable  equations  with  no  time 
delay.   Hess  [25]  has  extended  this  procedure  to  a  system 
in  which  the  delay  was  not  restricted. 

Results  similar  to  those  obtained  by  Ross  and  Fliigge- 
Lotz  [6]  were  developed  by  Soliman  and  Ray  [26,27]  by 
approximating  the  delay  with  an  arbitrarily  large  set  of 
differential  equations.   Similar  procedures  have  been  used 
by  Westdal  and  Lehn  [28]  for  a  related  problem. 
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II.   THEORY  OF  THE  OPTIMAL  CONTROL  OF 
DELAYED-DIFFERENTIAL  SYSTEMS 

A.   EXISTENCE  AND  UNIQUENESS 

For  an  equation  of  the  following  form, 

x(t)  =  f(t,x(t),x(t  -t)) 
x(t)  =  <j>(t)  ,   tQ  -  t  <   t  <_   tQ 
x(t,x(t))  >  0 

El'sgol'ts  [15]  has  shown  that  if  the  functions  f,  <$> ,    t  are 
continuous,  the  solution  exists.   If  in  addition 


8f 


9(x(t)) 


3f 


S(x(t  -T)) 


d<t>(t) 


dt 


and 


dx 
dt 


are  bounded  in  the  vicinity  of  the  initial  value  (a  Lipschitz 
condition  in  x(t)),  then  the  solution  is  also  unique. 

Since  it  has  been  shown  [6]  that  the  optimal  control, 
u*(t) ,  is  of  the  form 


?~lp  t 


0 


u*(t)  =  -R"1C'[K0x(t)  +   /  K1(8)x(t  + 6)d0] 

-T 

the  solution  to  the  optimal  regulator  problem  (equations  1-3) 
exists  and  is  unique  if  the  above  Lipschitz  condition  is 
satisfied. 
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B.  CONTROLLABILITY 

L.  Weiss  [29]  developed  algebraic  sufficient  conditions 
for  delayed-differential  equations  with  time  varying  coeffi- 
cients to  be  controllable.   These  conditions  reduce  to  the 
ordinary  algebraic  criteria  for  controllability  as  the  delay 
becomes  negligible.   The  paper  presents  a  method  for  computing 
the  columns  for  the  controllability  matrix. 

Considering  a  system  with  constant  coefficients,  Hewer 
[30]  has  shown  that  if  the  system  in  which  x-*-0  is  control- 
lable, then  the  delayed  equation  is  controllable  for  any 
t  >_  0.  The  converse  of  this  statement  is  not  always  true. 
For  the  time  invariant  case,  the  results  obtained  by  Weiss 
can  be  reduced  to  those  obtained  by  Hewer. 

C.  DEVELOPMENT  OF  THE  OPTIMAL  CONTROL  LAW 

Ross  and  Flugge-Lotz  [6]  and  Ross  [21]  have  developed 
the  optimal  control  law  for  equation  (1)  subject  to  (3)  as 
follows ; 

1.  Admissible  Control 

If  (1)  is  controllable,  the  set  of  admissible  control 
functions,  u(t),  are  those  -aoaaurcable.  functions  in  Lp[0,°°] 
which  yield  a  finite  value  for  (3)  for  any  set  of  initial 
conditions  <Kt). 

2.  Sufficient  Conditions  for  a  Linear  Feedback 
Control  Law  to  be  Optimal 

The  optimal  control  law  of  (1)  subject  to  the  cost 

function  (3)  and  a  specified  set  of  initial  condition 

functions  (2)  is  of  the  form 
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1  ° 

u*(t)  =  -R"xCt[K0x(t)  +   /  K1(6)x(t  + e)d6]         (5) 

— T 

t  _>  0  ;   x(t)  =  <j>(t)  ,   -t  <   t  <   0 


if 

a.  u*(t)  is  a  stable  control  law  (admissible  in  this  case) 

b.  KQ  (a  symmetric  positive  definite  matrix),  K-.  ( 6 )  and 
K2(C, 9)  satisfy  the  following  relationships 

A'KQ  +  KQA  -  K0CR-1C'K0  +  ^ '  ( 0 )  +  1^(0)  +  Q  =  0      (6) 


dK-^6) 

~ =  (A'  -  K0CR"1C,)K1(9)  +  K2(0,9)     -t  < 9  < 0   (7) 


30^ +  9^ =  -K1'(C)CR"1C'K1(9)  (8) 

-t  <  C  <  0  ,   -T<6<0 


K-^-t)  =  KQB  (9) 


K2(-t,9)  =  B'K1(6)  (10) 

If  B  =  0,  equivalent  to  a  system  with  no  delay  term,  the 
above  conditions  reduce  to  the  familiar  algebraic  Riccati 
equation.   The  first  term  in  u*(t)  is  the  optimal  control 
law  for  the  undelayed  system;  the  second  and  more  complicated 
term  accounts  for  the  addition  of  the  delay  term. 
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Under  the  above  conditions,  the  quadratic  cost  function 
(3)  can  be  expressed  in  terms  of  the  initial  condition 
function  (2). 

0 

j(4>,u*)  =  <t>,(o)K04)(o)  +  24>'(o)  /  K1(e)4.(e)de 

— T 

0   0 

+  /  J  <M(c)K2(c,9)<Ke)dcde  (li) 

-T  -T 

Although  the  control  law,  u*(t),  is  optimal  for  an  arbitrary 
initial  condition  function  which  is  continuously  differenti- 
able,  the  value  of  (3)  is,  as  would  be  expected,  a  direct 
function  of  the  initial  conditions. 

3.   Stability  of  the  Optimal  Control  Law 

In  order  to  apply  the  optimal  control  law,  some 
approximate  method  of  computation  is  required.   Once  the 
control  law  is  calculated,  there  is  no  known  algorithmic 
method  by  which  the  stability  of  the  control  can  be 
assessed  [6].   Determination  of  stability  requires  that  the 
closed  loop  system  performance  be  investigated  for  a  set  of 
initial  condition  functions  large  enough  to  cover  those 
expected  to  be  encountered  by  the  control  system.   If  no 
unstable  behavior  is  detected,  the  control  law  can  be 
assumed  to  be  stable  in  that  particular  application. 


17 


III.   APPROXIMATION  OF  THE  OPTIMAL  CONTROL  LAW 
FOR  DELAYED-DIFFERENTIAL  SYSTEMS 

A.   APPROXIMATION  OF  DELAYED-DIFFERENTIAL  EQUATIONS 

Repin  [23]  has  shown  that  equation  (1)  can  be  approxi- 
mated by  a  series  of  equations,  determined  by  defining  a  new 
set  of  state  variables.   The  state  variables  are  defined  as 
follows : 


x1«(t)  =  x(t) 


x2*^  =  x^  ~  e*t> 


x  *(t)  =  x(t  -  (i"1)T 
1  m-1 


(12) 


xm*(t)  =  X(t  -  t) 


Where  m  is  a  positive  integer  equal  to  or  greater  than  two, 
x.*(t)  is  a  n*l  column  vector  composed  of  the  elements  of 
the  original  state  vector. 

Using  simple  forward  difference  formulas  for  estimating 
derivatives,  the  following  set  of  equations  can  be  developed 
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x]_*(t)  =Axx*(t)  +Bxm*(t)  +Cu(t) 


x2*(t)  =  (Xl*(t)  -  x2*(t))  S=l 


V(t)  *<xi-l(t)  "  *if<t»  ^ 


(13) 


**•<*>-  (xm-l^  "  V<*»  ¥ 


This  series  of  first  order,  linear,  matrix  differential 
equations  can  be  shown  to  be  expressible  as 


y(t)   =  AQy(t)  +CQu(t) 


(14) 


where 


A 
(m-1) 


0 


0 
(m-1) 

T 

(m-1) 


B 


-(m-1) 


I  ...  0 
-(m-1) 


(15) 


Q, 


Q   0   •  •  •   0 

0   0   •  •  •   0 

0   0   •  •  •   0 


(16) 
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y(t)  e 


xx*(t) 
x2*(t) 


x±*(t) 


m 


(17) 


Repin  has  shown  that  by  selecting  m  large  enough, 
equation  (1)  can  be  approximated  to  any  degree  of  accuracy 
desired,  and  in  the  limit  as  m  -»■ «  ,   the  representation 
presented  above  becomes  exact. 

B.   OPTIMAL  AND  SUBOPTIMAL  CONTROL  OF 
ORDINARY  DIFFERENTIAL  SYSTEMS 

For  the  time  invariant  linear  system  of  differential 

equations 

x(t)  =  AQ  x(t)  +  CQ  u(t)   ;    t  >  0  ;   x(0)  =  <J> 

with  a  quadratic  cost  function  in  the  form  of  (3),  the  opti- 
mal control  law  has  been  shown  to  be 


u*(t)  =  -R  1  CQ'  V  x(t) 


(18) 


where  V  is  the  symmetric  positive  definite  solution  to  the 
algebraic  Riccati  equation 


VA0  +  AQ'  V  -  VCqR^Cq'V  +  QQ  =  0 


(19) 
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When  the  optimal  control  law  is  applied,  the  value  of 
the  cost  function  (3)  can  be  expressed  as 

J  =  <J>!  V<J> 

Elkind  and  Falb  [31]  have  shown  that  if  some  control 
u(t)  =  -Gx(t)  (G  is  a  matrix  of  constants  conformal  to  x(t)) 
is  applied,  instead  of  the  optimal  control  law  (18),  the  cost 
function  can  be  expressed  as 

J0  =  ♦'  Vo*  -  ♦'  V*  (20) 

further,  if  the  eigenvalues  of  (AQ  -  CQ  G)  have  negative  real 
parts,  VQ  is  the  symmetric  positive  definite  solution  to 

(AQ  -  CQ  G)'VQ  +  V0(AQ  -  CQ  G)  +  G'  R"1  G  +  QQ  =  0  (21) 

These  expressions  can  be  used  to  determine  the  value  of 
(3)  for  any  stable  feedback  composed  of  a  linear  combination 
of  states. 

C.   APPROXIMATION  OF  THE  CONTROL  LAW 

It  has  been  shown  that  (1)  can  be  approximated  by  a 
system  of  differential  equations 

y(t)  =  AQy(t)+  cQu(t)  ;  t  >  0  (22) 

y(t)  =  $  ;   t  =  0 

with  y(t),  y(t),  AQ,  and  CQ  as  defined  by  (12-17).   The 
initial  condition  matrix  is 
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(J,  E 


♦  (0) 


m-l 


♦  (-t) 


(23) 


The  cost  function  is 


J(y(0),y(t),u(t))  =  /  [yf(t)Qny(t)  +  u»(t)Ru(t)]  dt 

0        u 


(24) 


The  hypothesis  here  is  that  an  accurate  approximation  to 
the  optimal  control  law  (5)  for  the  delayed-differential 
system  (1)  can  be  obtained  from  the  optimal  control  lav:  (18) 
for  an  ordinary  differential  system.   This  ordinary  differen- 
tial system  is,  of  course   (22),  the  one  which  approximates 
the  delayed-differential  system  (1). 

Rigorous  proof  of  this  hypothesis  is  difficult.   Instead, 
the  following  section  shows  that  this  approximate  method  is 
exactly  equivalent  to  the  computational  approximation  which 
Ross  utilizes  in  solving  for  his  optimal  control  law  for 
system  (1). 

D.   EQUIVALENCE  OF  COMPUTATIONAL  APPROACHES* 
1.   Ross's  Method 

In  order  to  compute  the  optimal  control  law  (5), 
Ross  [21]  replaces  the  conditional  equations  (6-10)  by 


*For  convenience,  and  without  loss  of  generality,  the 
time  delay  t  has  been  normalized  to  unity  in  the  discussion 
which  follows. 
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finite  difference  formulas  on  the  surface  -1  <  9  <  0  , 
-1  <  £  <  0  .   The  resulting  equations  are 


AfK0   +   KQ  A-   KQ  CR"1  C  KQ   +   K^  (0)    +   K1(0)    +   Q   =    0 


(25) 


W=y#] 


-    K. 


(     -i 

m-1 


}  (srr)  ■  {A'  -  KoCR_lc'}Ki 


f-(l-l)l 


I    m-1    J 


+   K. 


L  "^     J 


1   <    i   <   m-1 


(26) 


M 


-(1-1)         -(J-i)] 


m-1 


m-1 


-   K, 


f -i  -(J-l)} 

m-1     ■        m-1 


+    K, 


f-(i-l)         -(3-1)1 

m-1       '         m-1 


-   K, 


f-(    1-D 

m-1 


-J 
•   m-1 


)  (W) 


*l' 


f-(i-l)l 


m-1 


C  R"1  C  K- 


f-(j-D) 


m-1 


1   <    i   <_  m-1 
1   <    j    <   m-1 


(27) 


K1(-l)    =   KQB 


K2    (-1  . 


.zl 


m-1 


=   B'  K. 


'.=1/ 


m-1 


(28) 


,      0   <    J    <   m-1  (29) 


with  A,  B,  C,  Q,  and  R  the  same  matrices  appearing  in  (1)  and 

(3). 
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Now  for  the  same  value  of  the  integer  m,  let  K  be  the 
positive  semidefinite  solution  of  the  algebraic  Riccati 
equation 

A'  ,  K  +  K  k      .    -K  C   .  R"1  C'  ,  K  +  Q   .  =  0     (30) 
m-1       m-l     m-1     m-1      m-1 


where 


lm-l 


A         0         0 
(m-l)I   -(m-l)I      0 
0       (m-l)I   -(m-l)I 


•    •    • 


B 
0 
0 


(m-l)I   -(m-l)I 


Q 


m-1 


0 
0 


'm-1 


(3D 


(I  is  the  n  x  n  identity  matrix,  A  -.    and  Q  ,  are  square 

J  'm-1      m-1 

matrices  having  n  x  m  rows  and  columns,  and  C   -,  is  n  x  m  by  I) 

If  one  considers  the  (symmetric)  matrix  K  to  consist  of 

p 
m  sub-blocks,  K.  .  ,  for  0  <_  i  <_  m-1,  and  0  <_  j  <_  m-1  ,  of 


n  x  n 


K  = 


K0,0    K0,l 
Kl,0    Kl,l 


K 


K 


K 


0,m-l 


l,m-l 


K 


(32) 


m-1       *   *   *      "m-l, m-1 
then  (30)  becomes  a  set  of  algebraic  equations  in  the  unknowns 

K, 


'«" 
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Ross  then  points  out  that  if  one  dissects  (30)  and 
compares  those  algebraic  relations  with  the  algebraic  rela- 
tions (25-29),  the  two  sets  of  equations  are  found  to  be  in 
direct  correspondence.   Specifically: 


K0  =  K0,0 


(33) 


K. 


-i 


T^=T) 


=  (m-1)  K 


0,i+l 


Kl(-1)  =  K0B  =  K0j0B 


0  <  i  <  m-2 


(34) 


M*^  -  T^j  =  (m-1)2Ki+l,j  +  l'   0<i<m-2  and 

0  <  j  <  m-2 


K, 


-1 


T^TJ 


=  B'  K. 


Jll 


H(m-D 


0  1J  lm-1 


K, 


-i 


[Tm-iy  » 


-1 


Ki' 


-i 


Tm^lT 


B 


0  <  i  <  m-1 


(35) 


Thus,  solution  of  the  Riccati  equation  (30)  is  equivalent 
to  solution  of  equations  (25-29).   Since  efficient  algorithms 
exist  for  the  solution  of  Riccati  equations  of  this  form, 
Ross  bases  his  computational  approach  on  the  solution  of 
equation  (30).   Interpolation  between  the  resulting  values 


of  K 


0 


K. 


-I 


m-1 


for  0  <  i  <  m-1   and   K. 


-l 


m-1  J  m-1 


_zi 


for 


i,j  =  0,1,...,  m-1  ,  yields  approximations  to  KQ,  K-,(  8 )  and 
Kp(c,6)  in  equations  (5)  and  (11).   The  computation  is 
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terminated  when  no  significant  improvement  in  the  approxi- 
mation of  KQ,  K-,(6),  Kp(£,9)  occurs  as  m  is  increased. 
2.   The  Method  of  Section  III.  C. 

It  has  been  shown  that  the  optimal  control  law  for 
(22)  with  a  cost  function  of  the  form  (2k)    is 


u«(t)  =  -R  1C0f  Vy(t) 


(36) 


where  V  is  the  solution  of  the  Riccati  equation 


V  V  +  VA0   "   VC0R-1C0'V  +   QQ   =    0 


(37) 


\  2  2 

V  is  a  symmetric  (n  *  m)   matrix  composed  of  m  sub-blocks 

V 


V  = 


11 


•     • 


V 


lm 


V 


ml 


mm 


(38) 


The  value  of  the  cost  function  J  is  given  by 


J  =  4>*  V  <J> 


(39) 


Equations  (36)  and  (39)  can  be  rewritten 


,-1 


m 


u*(t)  =  -R-XC0'  [Vuyi(t)  +  I   Vu  yi(t)] 


(40) 
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m 


J  =  ♦,(0)  V,,  <|>(0)  +  2  <fr'(0)  I     Vn,  * 
u  i=2   1:L 


rl-i^ 


m-1 


m   m   rn  4  ^ 

+  ■  I    w{ 


j=2  i=2 


V,  ,  $ 


[m-lj  'ij 


'izT 


m-1 


(4l) 


or 


m-1 


u*(t)  =  .r-iCo.  cv11?1(t)  t  J1  Xiui+i-^'W^CT  ] 

(42) 


m-1 


r  j  "\ 


J  = 


*,(0)Vn*(0)  +  2c(>'(0)  I   V1  i+1-(m-l)'4) 


-i 


m-1 


m-1 


m-1  m-1 


j=l  1=1 


m-1 


V 


l+l,J+l'(B|-1)  '♦ 


m-1 


(m-1) 


(43) 


Now  the  Integrals  in  the  optimal  control  law  and  cost 
function  of  equations  (5)  and  (11)  can  be  approximated  by 
finite  summations  as  follows: 


m-1 


u*(t)  *  -R"1  C  [Kn  x(t)  +  I      K, 

i=l   1 


-(i-1) 


m-1 


x 


t  - 


m-1 


m-1 


1  (44) 


m-1 
J  ^  $'(0)  KQ  <J>(0)  +  2<fr«(0)  I      K1 


i=l 


m-1 


f        .  ^ 


-i 


.m-1 


m-1 


m-1  m-1 


-l 


m-1 


K. 


-d-i)   -0-1)1 


2   m-1 


m-1 


Zl 


m-1 


1 


(m-1) 


(45) 
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Recalling  that  equations  (12)  and  (17)  state  that 

1-1 


y±(t)  =  x 


t  - 


m-1 


i  -  1 ,2 , . . .  jin 


then  each  term  in  equations  ( 42—43 )  corresponds  to  a  term  in 
(44-45).   This  correspondence  strongly  suggests 


vi,i+i  -  Ki 


V        =  K 
vi+l,j+l   A2 


f-(i-D) 


m-1 


m-1 


f-(i-l)    -(J-l)l  .  1 


m-1 


m-1 


(m-1) 


(46) 


Assuming  the  validity  of  equations  (46),  one  can  now  approxi- 
mate the  optimal  control  law  for  delayed-differential 
equations  by  utilizing  the  optimal  control  formulation  for 
ordinary  differential  systems. 
3.   Equivalence 

The  validity  of  equations  (46)  and  their  equivalence 
to  Ross's  solution  can  be  proven  by  noting  that  the  Riccati 
equation  (30)  which  Ross  utilizes  to  generate  his  solution  is 
identical  to  the  Riccati  equation  (37)  used  to  generate  the 
differential  solution. 

E.   ADVANTAGES  OF  THE  DIFFERENTIAL  APPROACH 

There  are  certain  advantages  to  be  gained  by  using  the 

differential  approximation  scheme  in  calculating  the  optimal 

control  law  for  delayed-differential  systems. 

First,  the  closed  loop  eigenvalues  of  the  approximating 

differential  system  yield  information  about  the  quality  of 
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the  differential  approximation,  and  about  the  stability 
characteristics  of  the  actual,  optimally  controlled  delayed- 
differential  system.   As  Hess  [25]  points  out,  the  largest 
closed  loop  damped  natural  frequency  of  the  differential 
system  should  be  small  when  compared  to  the  frequency  of 
occurrence  of  the  station  points  used  in  defining  x~*(t) 
through  x  *(t)  in  (13).   This  means  ensuring  that 


0), 

^ <<  1.0        j=l,2,.  .  .  ,nxm        (47) 


2TT(m-l) 


where  co.  is  the  magnitude  of  the  imaginary  part  of  the  j 
closed  loop  eigenvalue.   The  smaller  the  left  hand  side  of 
equation  (47),  the  more  accurate  are  the  forward  difference 
expressions  for  x?*(t)  through  x^Ct)  in  (13).   If  the 
restrictions  of  equation  (47)  are  met,  the  closed  loop 
eigenvalues  of  the  differential  system  give  excellent 
information  regarding  the  stability  characteristics  of  the 
actual,  optimally  controlled  delayed-differential  system. 

Second,  the  differential  approach  is  amenable  to  systems 
with  time  varying  parameters.  This  means  that  one  should  be 
able  to  find  an  approximation  to  the  optimal  control  law  for 
a  delayed-differential  system  of  the  form 

x(t)  =  A(t)  x(t)  +  B(t)  x[t  -  i(t)]  +  C(t)  u(t) 

subject  to  a  cost  function  of  the  form 
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J(<J)(t),x(t),u(t))  =  J  [x'(t)  Q(t)  x(t)  +  u*(t)  R(t)  u(t)]dt 

0 

by  utilizing  the  theory  of  optimal  control  of  ordinary  time 
varying  differential  systems. 

F.   ALGORITHM  FOR  RESTRICTED  FEEDBACK 

1.  Introduction 

The  optimal  control  law  (5)  may  require  a  digital 
computer  for  implementation.   The  question  naturally  arises 
as  to  just  how  well  the  system  (1)  would  perform  with  a 
simpler,  suboptimal  control  law.   For  example,  for  systems 
in  which  the  time  delay  is  relatively  small,  a  control  law 
of  the  form 

u(t)  =  -G1  x(t)  (48) 

may  yield  satisfactory  performance,  and  be  much  easier  to 
implement  than  control  law  (5).   If  law  (48)  is  not 
satisfactory 

u(t)  =  -G^Ct)-  G2  x(t-l)  (49) 

or 

u(t)  =  -G1x(t)-  G2  x(t-%)  -  G3x(t-1)  (50) 

might  be  considered.   The  next  section  offers  a  direct 
constructive  procedure  for  determining  the  G.  in  (48-50) 
based  upon  the  optimal  formulation  (5). 

2.  Restricted  Feedback 

The  method  to  be  described  is  based  upon  the  same 
linear  approximations  for  the  states  of  the  delayed-differential 
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system  as  used  in  the  differential  approximation  (13).  The 
procedure  involves  representing  x(t+6)  in  (5)  by  any  number 
of  straight  line  segments  as  follows: 

Zeroth  approximation 

x(t+e)  s:  x(t)   ,   0-rr^=^  -2^.6>.^:# 

First  approximation 

x(t+8)  s»  x(t)  +  [x(t)  -  x(t-l)]e  ,   o^e<_-l 

Second  approximation 


[X(t)  -x(t-»i)]e  ^ 


h 


-'A  i  s  -  ° 


{t+e)*  x(t-*)  +  [x(t^)-x(t-l)3(e+!s)  f  ^^^-I^G^   "^ 


Third  approximation 


(t+e)  s  x(t)  +  [x(t)-^x(t-V3)]e   ?   %^^g^  .-'/s^frO 


(t+e)*  x(-t-V3)  +  [xCt-yO-xCt-VOjce+i^)  , 


5? 


%  ^/3f^ 


(t+e)  s  x(t-V3)  +  [x(t"2/\)rx(t'1)](e+2/3)  , 


W 


=5^^^^--^ 
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A  simplified  control  law  based  upon  any  one  of  these 
approximations  can  be  computed  by  substituting  one  of  the 
above  expressions  for  x(t+e)  into  the  integral  expression 
in  the  optimal  control  law  (5).   For  example,  using  the 
second  approximation 


l  ° 

u*(t)  «  -R^C  [KQ  x(t)  +   J  K,  (e){x(t)  +2[x(t)  -x(t-Js)]6}  d6 

-h 

+  /  K1(e){x(t-j2) +2[x(t-jj) -x(t-i)](e+j5)}de] 


which  yields 


-l  r  ° 

u*(t)  ~  -R  X  C  [K,+  j    (1  +  26)  K1(6)  del  x(t) 

0  -h 

+  [-  /  26  K1(6)d6+.   /  (2  +  26)  K,  (0)  d6]  x(t-Ss) 
-h  -1  X 


+  [-  /  (1 +26)  K1(6)  d6  ]  x(t-l)| 


-^   -G1x(t)-  G2  x(t-^)  -  G-x(t-l) 


Since  KQ  and  K, (6)  have  already  been  calculated,  the  expres- 
sions above  can  be  numerically  integrated  to  yield  a 
simplified,  suboptimal  control  law. 

The  performance  of  the  delayed-differential  system 
with  any  of  these  suboptimal  control  laws  depends,  of  course, 
upon  how  well  the  straight  line  approximations  represent 
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x(t+8).   In  this  light,  if  a  suboptimal  law  based  upon  the 
zeroth  approximation  yields  a  significantly  larger  value  of 
the  cost  function  than  the  optimal  law,  a  higher  order 
approximation  (rather  than  a  different  value  of  G-,  )  is 
needed.   The  large  increment  in  the  cost  function  is 
indicating  that 

x(t  +  9)  •»   x(t)  ;      a^-gr^rrr  ~  I  ^.  ©"  —  C 


is  not  a  satisfactory  approximation. 

3.    Stability  and  Cost  Function  Evaluation 

Information  about  the  stability  of  the  suboptimal 
control  law  can  be  obtained  from  the  closed-loop  eigenvalues 
of  the  differential  system  as  outlined  in  Section  III.  E. 
The  cost  function  can  be  evaluated  either  directly  from 
simulation  or  in  approximate  fashion  by  using  expressions 
similar  to  (20-21). 
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IV.   SAMPLE  CONTROLLER  DEVELOPMENT 


A.   SYSTEMS  TO  BE  STUDIED 
The  scalar  equations 


x(t)  =  a11x(t)+  b11x(t-l)  +  c11  u(t) 


x(t)  =  l+t,   -1  <  t  <  0 


(51) 
(52) 


J  «  /  (x2(t)  +  u2(t))dt 
0 


(53) 


were  studied  for  a,1  =  b,-.  =  -c,,  =  -1  (Case  1)  and 

*  * 

a.,.,  =  b,,  =  -c-,-,  =  -2  (Case  2). 

For  Case  1,  the  required  order  of  the  differential 
approximation  was  determined  to  be  ten;  for  Case  2,  the  order 
was  determined  to  be  fourteen.   These  values  for  m  insured 
that  a>./(2Tr(m-l))  ^  .1  ,  j  =  l,2,...,m 

The  differential  approximations,  expressed  in  matrix 
notation  are: 

y(t)  =  AQ  y(t)  +  Cn  u(t)   ,   t  >0  (5^) 

y(0)  =  ? 


'0 


J   =   /  (y'(t)  Qny(t)  +  iT(t))dt 
0         u 


all 
(m-1)   -(m-1) 
0      (m-1) 


•    •    • 


ull 
0     0 
0      0 

(m-1)   -(m-1) 


(55) 

(56) 


(57) 


3^ 


V  =  [cll  °  •  •  •  o  o] 


<t>»   =  [1   (m-2)/(m-l)   .  .  .   (m-l-i)/(m-l)  .  .  .  0]  , 


Q, 


.  o 

.   0 


B.   FORMULATION  OP  THE  OPTIML  CONTROL  LAW 

The  optimal  control  law  for  the  delayed-dif ferential 
system  (51-53)  was  obtained  by  utilizing  the  differential 
approximation  (5^-56).   The  matrix  Riccati  equation  (19) 
for  the  differential  approximation  was  solved  and  the  resulting 
V..  used  to  determine  KQ  and  K-,(6)  via 


K0  =  Vll 


K 


l(m-lj 


(m-1) V 


l,i+2 


1=0,1,2,. . .,(m-2) 


K-,(-l),  not  normally  available  from  the  differential  approach, 
was  obtained  from  Ross's  solution  (equation  (3*0)  as 


Kl(-1)  =  bllK0 


The  resulting  Kn ,  K-.(8)  and  cost  function  values  are  shown 
in  Figures  1  and  2  for  Cases  1  and  2  respectively.   Figure  3 
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shows  x(t)  for  the  uncontrolled  and  optimally  controlled 
delayed-differential  system  for  Case  1.   Figure  4  presents 
the  optimal  control  function,  u*(t),  for  Case  1.   Figures 
5  and  6  show  similar  results  for  Case  2. 

C.   FORMULATION  OF  A  RESTRICTED  FEEDBACK  CONTROL  LAW 

Suboptimal  control  laws  were  developed  for  feedback  of 
a  restricted  set  of  state  variables  associated  with  the 
differential  approximation  (51-57)  by  the  method  presented 
in  III.  F.2.   These  control  laws  are: 

Zeroth  approximation 
Case  1 

u(t)  =  -.216  x(t) 
Case  2 

u(t)  =  -.196  x(t) 
First  approximation 
Case  1 

u(t)  =  -.3l0x(t)+  .123x(t-l) 
Case  2 

u(t)  =  -.409x(t)+  .213x(t-l) 
Second  approximation 
Case  1 

u(t)  =  -.384x(t)+  .0887x(t-^)+  .079x(t-l) 
Case  2 

u(t)  =  -.i*M78x(t)+  .115x(t-^+  .155x(t-l) 
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Third  approximation 
Case  1 

u(t)  =  -.392x(t)+  .0384  x(t-.33)  +  .0803  x(t-.  67) 

+  .0570  x(t-l) 
Case  2 

u(t)  =  -.i»*5x(t')+  .024  x(t-.33)  +  .128x(t-.67) 
.+  .115  x(t-l) 

The  value  of  the  cost  function  vs.  the  number  of  states 
in  the  restricted  feedback  is  presented  in  Figures  7  and  8 
for  Cases  1  and  2,  respectively.   Figure  9  presents  x(t)  for 
the  delayed-differential  system  of  Case  1  with  no  control 
and  with  the  control  law  determined  from  the  third  approxi- 
mation above.   The  control  function  for  the  third  approxima- 
tion is  shown  in  Figure  10.   Figures  11  and  12  present 
similar  results  for  Case  2.   In  both  cases,  the  use  of  a 
fourth  state  in  the  feedback  (third  approximation)  produced 
less  than  a  one  percent  improvement  in  the  value  of  the  cost 
function. 

In  order  to  compare  the  restricted  feedback  gains 
obtained  by  the  algorithm  of  Section  III.F.2  with  optimal 
gains  associated  with  feedback  composed  of  one  and  two 
states,  a  gradient  optimization  technique  was  mechanized. 
In  this  technique,  equations  (20)  and  (21)  were  solved 
recursively  to  yield  the  optimum  values  of  the  restricted 
feedback  gains  for  the  differential  systems  of  Cases  1  and  2. 


A  more  efficient  method  for  accomplishing  this  opti- 
mization can  be  found  in  Ref.  32. 
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Case  1 


A  check  was  then  made  to  ensure  that  these  gains  were  the 
optimal  restricted  feedback  gains  for  the  actual  delayed- 
differential  system  by  direct  integration  of  equations  (51) 
and  (53).   The  results  are 
Case  1 

u(t)  =  -.300  x(t) 
J  =  .39240 
Case  2 

u(t)  =  -.500  x(t) 
J  =  .3729 

u(t)  =  -.330x(t)+  .130x(t-l) 
J  =  .37764 

u(t)  =  -.400x(t)  +  .225x(t-l) 
J  =  .3314 

Comparison  of  these  results  with  those  obtained  from  the 
zeroth  and  first  order  approximations  indicate  that  the 
algorithm  was  moving  the  restricted  feedback  gains  in  the 
right  direction  as  the  order  of  the  approximations  was 
increased.   The  utility  of  the  algorithm  is  also  borne  out 
by  Figures  7  and  8. 


Case  2 
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V.   CONCLUSIONS 

This  research  has  shown  that  the  optimal  control  law 
for  a  delayed-differential  system  can  be  determined  to  any 
desired  accuracy  from  the  optimal  control  law  for  an  ordi- 
nary differential  system.   Since  the  theory  associated 
with  differential  systems  is  well  established,  and  efficient 
algorithms  for  the   required  computations  are  readily  avail- 
able, this  approach  has  obvious  merit. 

Further,  it  has  been  demonstrated  that  a  simple  algorithm 
can  be  used  to  determine  a  suboptimal,  restricted  feedback 
control  law. 

The  closed  loop  eigenvalues  of  the  differential  approxi- 
mation can  be  used  to  yield  information  about  the  conver- 
gence of  the  approximation.   Although  these  eigenvalues 
completely  determine  the  stability  of  the  differential 
approximation,  they  can  only  suggest,  albeit  strongly,  that 
the  resulting  control  law  will  produce  a  stable  delayed- 
differential  system.   Further  research  is  required  to 
ascertain  the  conditions  under  which  a  control  law  determined 
by  this  method  will  definitely  produce  a  stable  delayed- 
differential  system. 
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